{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aa6f0b77",
   "metadata": {},
   "source": [
    "## **Introduction to ADCNet: Predicting ADC Activity with DeepChem**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f2f37d4",
   "metadata": {},
   "source": [
    "Advancements in molecular biology have revolutionized drug discovery, focusing on more selective clinical candidates. Traditional small-molecule inhibitors face limitations like off-target toxicity and drug resistance. Monoclonal antibodies (mAbs) improved targeting accuracy but still struggle with heterogeneous cancer cell populations. To address these issues, Antibody-Drug Conjugates (ADCs) have emerged as a promising cancer therapy. By combining the specificity of mAbs with potent cytotoxic drugs linked through chemical linkers, ADCs enable targeted delivery to cancer cells, reducing harm to healthy tissues. However, developing effective ADCs is complex, requiring careful selection of antibodies, payloads, and linkers, which all impact safety and efficacy.\n",
    "\n",
    "Therefore, ADCNet [[1]](#1) has been developed as a comprehensive deep learning framework that addresses the above issues and accurately predicts the activity of antibody-drug conjugates. It utilizes ESM-2 for key features from antibody and antigen sequences and FG-BERT for processing the SMILES strings of linkers and payloads. The framework also incorporates the Drug-Antibody Ratio (DAR), enhancing the rational design of safer and more effective ADC candidates. By learning and integrating patterns from these various molecular components, ADCNet aids in the rational design of safer and more effective ADC candidates. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d633c53",
   "metadata": {},
   "source": [
    "In this tutorial, we will explore how to predict the therapeutic activity of Antibody-Drug Conjugates (ADCs) using ADCNet, a unified deep learning framework implemented in DeepChem. Before, proceeding with this guide, it is recommended to build a foundational understanding of ADCs. Refer to the \"Introduction to Antibody-Drug Conjugates\" [[2]](#2) notebook available in the DeepChem tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c8451a",
   "metadata": {},
   "source": [
    "# **Colab**\n",
    "This tutorial and the rest in this sequence can be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/Introduction_to_ADCNet.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93b88c1d",
   "metadata": {},
   "source": [
    "## **Evolution of ADCs**\n",
    "\n",
    "As the field continues to advance, the design of ADCs has been progressively refined over several generations. First-generation ADCs, such as *gemtuzumab ozogamicin*, were built using humanized antibodies, a single type of cytotoxic drug, and acid-sensitive linkers. These components were originally derived from murine antibodies and conventional chemotherapy agents. The conjugation process was typically random, targeting lysine or cysteine residues, which resulted in mixtures with uneven Drug-Antibody Ratios (DAR). This lack of uniformity made it difficult to establish a consistent therapeutic index and often led to off-target toxicity and narrow treatment windows. As a result, first-generation ADCs were prone to unmanageable side effects and limited clinical effectiveness.<br>\n",
    "\n",
    "In contrast, second-generation ADCs, such as *brentuximab vedotin* and *trastuzumab emtansine*, introduced more powerful cytotoxic agents like tubulin inhibitors, along with more stable linkers. These improvements significantly enhanced both the treatment’s efficacy and molecular stability.<br>\n",
    "\n",
    "Moreover, the third generation ADCs, represented by drugs like *polatuzumab vedotin* and *enfortumab vedotin*, introduced site-specific conjugation techniques and hydrophilic linkers. These innovations allowed for precise control over DAR, thereby enhancing both safety and therapeutic efficacy. [[3]](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ab8ddba",
   "metadata": {},
   "source": [
    "Let's take a closer look at the evolution of Antibody-Drug Conjugates (ADCs) with some of the first, second, and third-generation ADCs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ad32e1",
   "metadata": {},
   "source": [
    "<img src = \"assets/generations_adc.jpg\" alt=\"image\" height=\"700\" width=\"800\"> <br> **Fig.1** Schematic representations of first, second, and third-generation Antibody-Drug Conjugates (ADCs). [[4]](#4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3401274c",
   "metadata": {},
   "source": [
    "As we analyze the evolution of ADC design through the generations, we can observe significant advancements in various aspects. The figure above demonstrates the progression of ADC design through various generations. It highlights key differences, including linker cleavability (cleavable versus non-cleavable), the format of the monoclonal antibody (such as the IgG1 subtype), and the number of cytotoxic warheads attached to each antibody. Each generation shows advancements in specificity, stability, and therapeutic efficacy, marked by improvements in site-specific conjugation and payload delivery mechanisms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "430fbc0b",
   "metadata": {},
   "source": [
    "Now that we have explored the fundamentals and evolution of ADCs, let's delve into the ADCNet architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bb6044",
   "metadata": {},
   "source": [
    "## **Overview of the model architecture**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36331efe",
   "metadata": {},
   "source": [
    "We follow a three-step execution process. First, we process different types of input data. Second, we generate embeddings from these inputs using pretrained models. Finally, we concatenate the embeddings and feed them into a Multilayer Perceptron (MLP) to make predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d2cebf7",
   "metadata": {},
   "source": [
    "### Inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d9cd28e",
   "metadata": {},
   "source": [
    "We examine three different types of inputs used in the model:\n",
    "\n",
    "(I) **Protein Sequences:**\n",
    "- **Antibody Heavy Chain:** The protein sequence of the antibody's heavy chain.\n",
    "- **Antibody Light Chain:** The protein sequence of the antibody's light chain.\n",
    "- **Antigen:** The protein sequence of the target antigen.\n",
    "\n",
    "(II) **Small Molecules (SMILES representations):**\n",
    "- **Linker:** A SMILES string representing the chemical structure of the linker.\n",
    "- **Payload:** A SMILES string representing the chemical structure of the cytotoxic payload.\n",
    "\n",
    "(III) **Numerical Value:**\n",
    "- **Drug–Antibody Ratio (DAR):** A numerical value indicating the average number of payload molecules attached to each antibody.\n",
    "\n",
    "Each input type is processed individually to extract its unique set of features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f956f172",
   "metadata": {},
   "source": [
    "### Generating Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37c21ff",
   "metadata": {},
   "source": [
    "ADCNet uses pre-trained language models to transform the inputs into embeddings:\n",
    "\n",
    "- Protein sequences (antibody heavy chain, antibody light chain, and antigen sequences) are processed using ESM-2 (Evolutionary Scale Modeling) [[4]](#4), a Transformer-based protein language model. ESM-2 converts these sequences into dense embeddings that encode their structural and functional properties.\n",
    "- SMILES representations of the linker and payload are processed using ChemBERTA, which generates embeddings that capture the chemical properties of these small molecules.\n",
    "\n",
    "(**Note**: While the original ADCNet paper utilized FGBERT, we are employing ChemBERTA here due to its availability and effectiveness within the DeepChem framework.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7347c06",
   "metadata": {},
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b753682f",
   "metadata": {},
   "source": [
    "After generating the embeddings:\n",
    "\n",
    "- The embeddings from the three protein sequences (heavy chain, light chain, and antigen), the two small molecules (linker and payload), and the processed DAR value are combined into a single feature vector. \n",
    "- This combined feature vector is then input into a Multilayer Perceptron (MLP), which consists of two fully connected layers with non-linear activation functions. The MLP processes these concatenated features to predict the therapeutic activity of the antibody-drug conjugate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1b7b0a",
   "metadata": {},
   "source": [
    "Below is the architecture diagram of ADCNet, illustrating the complete workflow from input sequences and molecular structures through embedding layers and model components to the final prediction output."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15f90a13",
   "metadata": {},
   "source": [
    "<img src=\"assets/ADCNet_2.png\" alt=\"image2\" height = \"800\" width=\"800\"> <br> **Fig.2** Diagram illustrating the network architecture of ADCNet model. [[4]](#4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b392de31",
   "metadata": {},
   "source": [
    "### Versatility of ADCNet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dc056aa",
   "metadata": {},
   "source": [
    "According to the original [ADCNet paper](https://arxiv.org/pdf/2401.09176), [[1]](#1) the architecture utilizes the plM ESM-2 to process antibody and antigen sequences. While ESM-2 is highly effective, confining the framework to a single model limits its adaptability to other protein representation techniques that could provide complementary strengths. Researchers can explore alternative models, such as ProtBERT, T5-Protein, or newer protein language models that are tailored to specific tasks. This flexibility enables experimentation with models that can better capture the structural and functional nuances of antibodies and antigens, depending on the context of the antibody-drug conjugate (ADC) design. For instance, some models might excel in predicting binding affinity, while others could be more skilled at managing sequence diversity.\n",
    "\n",
    "Similarly, the original ADCNet architecture utilizes FG-BERT for encoding linker and payload SMILES strings. Although FG-BERT is effective for small molecule representation, relying solely on it limits the framework's ability to incorporate advancements in chemical modeling. To enhance performance, researchers are encouraged to adopt other small molecule models, such as ChemBERTa, MolBERT, or various graph-based and transformer-based chemical encoders.\n",
    "\n",
    "In our implementation, we use ESM-2-8M, but researchers are encouraged to experiment with other ESM variants, such as ESM-2 650M and ESM-2-3B, which may offer improved accuracy. Additionally, we have replaced FG-BERT with ChemBERTa in this implementation to leverage its strengths in small molecule representation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aae7db8",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25e30b4e",
   "metadata": {},
   "source": [
    "Before we continue, let's install DeepChem in our environment and set up the other required dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "975c6ca8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install the necessary libraries\n",
    "!pip install deepchem numpy torch scikit-learn transformers tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e65167d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepchem as dc\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from tqdm import tqdm\n",
    "from transformers import EsmTokenizer, EsmModel\n",
    "from transformers import AutoModel, AutoTokenizer\n",
    "from deepchem.trans import NormalizationTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66558ea9",
   "metadata": {},
   "source": [
    "### Data Collection and Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d24b9644",
   "metadata": {},
   "source": [
    "We will be using ADCdb, which was originally utilized by ADCNet and can be accessed at [[5]](#5). This database contains information on 6,572 antibody-drug conjugates (ADCs), including 359 that are approved by the FDA or are currently in clinical trials, 501 in preclinical testing, 819 with in vivo testing data, 1,868 with cell line or target testing data, and 3,025 without such testing.\n",
    "\n",
    "For our purposes, we will use the preprocessed data for convenience, which is available at [[6]](#6). Let’s examine the dataset. The original ADCdb is a comprehensive collection of data on ADCs, and we will be working with the preprocessed version provided by ADCNet. This version can be found in the assets folder or in the ADCdb GitHub repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f43a6aef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have data of 435 ADCs.\n"
     ]
    }
   ],
   "source": [
    "# load file\n",
    "file_path = \"assets/adcdb.csv\"\n",
    "df = pd.read_csv(file_path)\n",
    "\n",
    "print(f'We have data of {len(df)} ADCs.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "efb5dcd2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['index', 'ADC ID', 'ADC Name', 'Antibody Name', 'Antibody Heavy Chain Sequence', 'Antibody Light Chain Sequence', 'Antigen Sequence', 'Payload Isosmiles', 'Linker Isosmiles', 'DAR', 'label（10nm）', 'label（100nm）', 'label（1nm）', 'label（1000nm）', 'DAR_val']\n"
     ]
    }
   ],
   "source": [
    "df.columns = [col.strip().replace(' (', '(').replace(') ', ')') for col in df.columns]\n",
    "print(df.columns.to_list())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aa112c7",
   "metadata": {},
   "source": [
    "We can see the dataset contains columns representing ADC names, antibody sequences, antigen sequences, SMILES strings for linker/payload, and labels at multiple concentrations. Now lets have a preview of the dataset we will be using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f2407728",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>ADC ID</th>\n",
       "      <th>ADC Name</th>\n",
       "      <th>Antibody Name</th>\n",
       "      <th>Antibody Heavy Chain Sequence</th>\n",
       "      <th>Antibody Light Chain Sequence</th>\n",
       "      <th>Antigen Sequence</th>\n",
       "      <th>Payload Isosmiles</th>\n",
       "      <th>Linker Isosmiles</th>\n",
       "      <th>DAR</th>\n",
       "      <th>label（10nm）</th>\n",
       "      <th>label（100nm）</th>\n",
       "      <th>label（1nm）</th>\n",
       "      <th>label（1000nm）</th>\n",
       "      <th>DAR_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>DRG0ABJAM</td>\n",
       "      <td>Trastuzumab-BCN-HydraSpace-Val-Cit-PABC-Gly-Ca...</td>\n",
       "      <td>Trastuzumab</td>\n",
       "      <td>EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLE...</td>\n",
       "      <td>DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKL...</td>\n",
       "      <td>MELAALCRWGLLLALLPPGAASTQVCTGTDMKLRLPASPETHLDML...</td>\n",
       "      <td>CCN(C(=O)CN)C1COC(OC2C(OC3C#C/C=C\\C#CC4(O)CC(=...</td>\n",
       "      <td>CC(C)C(NC(=O)OCCN(CCOC(=O)NC(C(=O)NC(CCCNC(N)=...</td>\n",
       "      <td>1.86</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>DRG0ZBATX</td>\n",
       "      <td>Anti-KIT NEG087?SSNPP-DM3</td>\n",
       "      <td>Anti-KIT mAb NEG087</td>\n",
       "      <td>EVQLVESGGGLVQPGGSLRLSCAASGFTFSDYYMAWVRQAPGKGLE...</td>\n",
       "      <td>DIQMTQSPSSLSASVGDRVTITCRASQSISSYLNWYQQKPGKAPKL...</td>\n",
       "      <td>MRGARGAWDFLCVLLLLLRVQTGSSQPSVSPGEPSPPSIHPGKSDL...</td>\n",
       "      <td>C[C@@H]1[C@@H]2C[C@]([C@@H](/C=C/C=C(/CC3=CC(=...</td>\n",
       "      <td>CC(S)CCC(N)=O</td>\n",
       "      <td>3.0-4.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>DRG0XJKXB</td>\n",
       "      <td>Trastuzumab-C239I-SG3400</td>\n",
       "      <td>Engineered trastuzumab</td>\n",
       "      <td>EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLE...</td>\n",
       "      <td>DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKL...</td>\n",
       "      <td>MELAALCRWGLLLALLPPGAASTQVCTGTDMKLRLPASPETHLDML...</td>\n",
       "      <td>C=C1CC2C=Nc3cc(OCCCOc4cc5c(cc4OC)C(=O)N4CC(=C)...</td>\n",
       "      <td>C[C@@H](C(=O)NC1=CC=C(C=C1)CO)NC(=O)[C@H](C(C)...</td>\n",
       "      <td>1.71</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>DRG0ZOYQV</td>\n",
       "      <td>Datopotamab deruxtecan</td>\n",
       "      <td>Datopotamab</td>\n",
       "      <td>QVQLVQSGAEVKKPGASVKVSCKASGYTFTTAGMQWVRQAPGQGLE...</td>\n",
       "      <td>DIQMTQSPSSLSASVGDRVTITCKASQDVSTAVAWYQQKPGKAPKL...</td>\n",
       "      <td>MARGPGLAPPPLRLPLLLLVLAAVTGHTAAQDNCTCPTNKMTVCSP...</td>\n",
       "      <td>CC[C@@]1(C2=C(COC1=O)C(=O)N3CC4=C5[C@H](CCC6=C...</td>\n",
       "      <td>C1=CC=C(C=C1)C[C@@H](C(=O)NCC(=O)O)NC(=O)CNC(=...</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>DRG0COMTY</td>\n",
       "      <td>Telisotuzumab vedotin</td>\n",
       "      <td>Telisotuzumab</td>\n",
       "      <td>QVQLVQSGAEVKKPGASVKVSCKASGYIFTAYTMHWVRQAPGQGLE...</td>\n",
       "      <td>DIVMTQSPDSLAVSLGERATINCKSSESVDSYANSFLHWYQQKPGQ...</td>\n",
       "      <td>MKAPAVLAPGILVLLFTLVQRSNGECKEALAKSEMNVNMKYQLPNF...</td>\n",
       "      <td>CC[C@H](C)[C@@H]([C@@H](CC(=O)N1CCC[C@H]1[C@@H...</td>\n",
       "      <td>CC(C)[C@@H](C(=O)N[C@@H](CCCNC(=O)N)C(=O)NC1=C...</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index     ADC ID                                           ADC Name  \\\n",
       "0      0  DRG0ABJAM  Trastuzumab-BCN-HydraSpace-Val-Cit-PABC-Gly-Ca...   \n",
       "1      1  DRG0ZBATX                          Anti-KIT NEG087?SSNPP-DM3   \n",
       "2      2  DRG0XJKXB                           Trastuzumab-C239I-SG3400   \n",
       "3      3  DRG0ZOYQV                             Datopotamab deruxtecan   \n",
       "4      4  DRG0COMTY                              Telisotuzumab vedotin   \n",
       "\n",
       "            Antibody Name                      Antibody Heavy Chain Sequence  \\\n",
       "0             Trastuzumab  EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLE...   \n",
       "1     Anti-KIT mAb NEG087  EVQLVESGGGLVQPGGSLRLSCAASGFTFSDYYMAWVRQAPGKGLE...   \n",
       "2  Engineered trastuzumab  EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLE...   \n",
       "3             Datopotamab  QVQLVQSGAEVKKPGASVKVSCKASGYTFTTAGMQWVRQAPGQGLE...   \n",
       "4           Telisotuzumab  QVQLVQSGAEVKKPGASVKVSCKASGYIFTAYTMHWVRQAPGQGLE...   \n",
       "\n",
       "                       Antibody Light Chain Sequence  \\\n",
       "0  DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKL...   \n",
       "1  DIQMTQSPSSLSASVGDRVTITCRASQSISSYLNWYQQKPGKAPKL...   \n",
       "2  DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKL...   \n",
       "3  DIQMTQSPSSLSASVGDRVTITCKASQDVSTAVAWYQQKPGKAPKL...   \n",
       "4  DIVMTQSPDSLAVSLGERATINCKSSESVDSYANSFLHWYQQKPGQ...   \n",
       "\n",
       "                                    Antigen Sequence  \\\n",
       "0  MELAALCRWGLLLALLPPGAASTQVCTGTDMKLRLPASPETHLDML...   \n",
       "1  MRGARGAWDFLCVLLLLLRVQTGSSQPSVSPGEPSPPSIHPGKSDL...   \n",
       "2  MELAALCRWGLLLALLPPGAASTQVCTGTDMKLRLPASPETHLDML...   \n",
       "3  MARGPGLAPPPLRLPLLLLVLAAVTGHTAAQDNCTCPTNKMTVCSP...   \n",
       "4  MKAPAVLAPGILVLLFTLVQRSNGECKEALAKSEMNVNMKYQLPNF...   \n",
       "\n",
       "                                   Payload Isosmiles  \\\n",
       "0  CCN(C(=O)CN)C1COC(OC2C(OC3C#C/C=C\\C#CC4(O)CC(=...   \n",
       "1  C[C@@H]1[C@@H]2C[C@]([C@@H](/C=C/C=C(/CC3=CC(=...   \n",
       "2  C=C1CC2C=Nc3cc(OCCCOc4cc5c(cc4OC)C(=O)N4CC(=C)...   \n",
       "3  CC[C@@]1(C2=C(COC1=O)C(=O)N3CC4=C5[C@H](CCC6=C...   \n",
       "4  CC[C@H](C)[C@@H]([C@@H](CC(=O)N1CCC[C@H]1[C@@H...   \n",
       "\n",
       "                                    Linker Isosmiles      DAR  label（10nm）  \\\n",
       "0  CC(C)C(NC(=O)OCCN(CCOC(=O)NC(C(=O)NC(CCCNC(N)=...     1.86            0   \n",
       "1                                      CC(S)CCC(N)=O  3.0-4.0            0   \n",
       "2  C[C@@H](C(=O)NC1=CC=C(C=C1)CO)NC(=O)[C@H](C(C)...     1.71            1   \n",
       "3  C1=CC=C(C=C1)C[C@@H](C(=O)NCC(=O)O)NC(=O)CNC(=...        4            1   \n",
       "4  CC(C)[C@@H](C(=O)N[C@@H](CCCNC(=O)N)C(=O)NC1=C...      3.1            1   \n",
       "\n",
       "   label（100nm）  label（1nm）  label（1000nm）  DAR_val  \n",
       "0             0           0              0     1.86  \n",
       "1             0           0              0     3.50  \n",
       "2             1           1              1     1.71  \n",
       "3             1           1              1     4.00  \n",
       "4             1           1              1     3.10  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb32ef4c",
   "metadata": {},
   "source": [
    "Set up the computation device (GPU if available, otherwise CPU) and import necessary transformer modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5fff0f8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose device\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d81af8ce",
   "metadata": {},
   "source": [
    "### Preprocessing Numeric Features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e258e4",
   "metadata": {},
   "source": [
    "We can see that the dataset includes an important feature: the Drug-Antibody Ratio (DAR), which plays a significant role in determining the efficacy and safety of ADCs, as it represents the average number of drug molecules attached to each antibody. Since DAR is a continuous numerical feature, we will scale it before inputting it into our model. Standardizing DAR to have zero mean and unit variance ensures that it is on a comparable scale with other features, which helps neural networks train more efficiently and converge faster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d461d53",
   "metadata": {},
   "source": [
    "Now we'll proceed to normalize them using DeepChem’s NormalizationTransformer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1c682d74",
   "metadata": {},
   "outputs": [],
   "source": [
    "dar_values = df['DAR_val']\n",
    "n_samples = len(dar_values)\n",
    "dar_dataset = dc.data.NumpyDataset(X=dar_values.values.reshape(-1, 1))\n",
    "normalizer = dc.trans.NormalizationTransformer(transform_X=True, dataset=dar_dataset)\n",
    "\n",
    "# Transform the dataset (this normalizes the X values to zero mean and unit variance)\n",
    "normalized_dataset = normalizer.transform(dar_dataset)\n",
    "\n",
    "# Extract the scaled DAR values\n",
    "dar_scaled = normalized_dataset.X.flatten()\n",
    "dar_scaled = torch.tensor(dar_scaled, device=device, dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "23159543",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original DAR mean: 3.8685, std: 1.5709\n",
      "Scaled DAR mean: -0.0000, std: 1.0012\n"
     ]
    }
   ],
   "source": [
    "print(f\"Original DAR mean: {dar_values.mean():.4f}, std: {dar_values.std():.4f}\")\n",
    "print(f\"Scaled DAR mean: {dar_scaled.mean():.4f}, std: {dar_scaled.std():.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "301eccb1",
   "metadata": {},
   "source": [
    "Now that our data is preprocessed and the Drug–Antibody Ratio (DAR) values are standardized, we are ready to generate embeddings using pre-trained models for each input type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f8e76c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract sequences and SMILES from the dataframe\n",
    "\n",
    "heavy_chains = df['Antibody Heavy Chain Sequence'].astype(str).tolist()\n",
    "light_chains = df['Antibody Light Chain Sequence'].astype(str).tolist()\n",
    "antigens = df['Antigen Sequence'].astype(str).tolist()\n",
    "linkers = df['Linker Isosmiles'].astype(str).tolist()\n",
    "payloads = df['Payload Isosmiles'].astype(str).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4005443",
   "metadata": {},
   "source": [
    "### Generating Embeddings with Pretrained Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e91703",
   "metadata": {},
   "source": [
    "Before we move further, it's necessary to know about ESM-2. ESM-2 (Evolutionary Scale Modeling) is a protein language model using a transformer-based architecture to process protein sequences. It has been trained on large datasets of protein sequences to learn the relationships between amino acids and the structural and functional properties of proteins.\n",
    "\n",
    "ESM-2 has demonstrated strong performance across various protein-related prediction tasks, making it a reliable choice for encoding protein sequences in deep learning workflows. To explore ESM-2 and other protein language models developed by Meta’s FAIR (Fundamental AI Research) team, visit the official GitHub repository [here](https://github.com/facebookresearch/esm)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c28b9c",
   "metadata": {},
   "source": [
    "Here we use the smallest ESM-2 model (esm2_t6_8M_UR50D, 6 layers, 8M parameters) for protein sequence embeddings. Larger ESM-2 models are available in the [Hugging Face Model Hub](https://huggingface.co/facebook/esm2_t6_8M_UR50D) for improved accuracy at the cost of increased computational resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d96a422",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load ESM-2 for protein sequences\n",
    "esm_model_name = \"facebook/esm2_t6_8M_UR50D\"\n",
    "tokenizer_esm = AutoTokenizer.from_pretrained(esm_model_name)\n",
    "esm_model = AutoModel.from_pretrained(esm_model_name).to(device)\n",
    "esm_model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f2f8fe",
   "metadata": {},
   "source": [
    "For small molecules like payloads and linkers, we will use ChemBERTa to generate embeddings from the SMILES strings of payloads and linkers. Chemberta is a transformer style model for learning on SMILES strings. The model architecture is based on the RoBERTa architecture and can be used for both pretraining an embedding and finetuning for downstream applications."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a642512d",
   "metadata": {},
   "source": [
    "Here, we are using the RoBERTa Featurizer from DeepChem alongside ChemBERTa from Hugging Face. The RoBERTa Featurizer, a wrapper for the RoBERTa Tokenizer in Hugging Face's Transformers library, efficiently tokenizes large SMILES string datasets for RoBERTa-based models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "837d4ff7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load ChemBERTa for molecular sequences\n",
    "from deepchem.feat import RobertaFeaturizer\n",
    "\n",
    "featurizer = RobertaFeaturizer.from_pretrained(\"DeepChem/ChemBERTa-77M-MLM\")\n",
    "model = AutoModel.from_pretrained(\"DeepChem/ChemBERTa-77M-MLM\")\n",
    "model.to(device)\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba6c9100",
   "metadata": {},
   "source": [
    "We have now initialized two Transformer-based models: ESM-2 for protein sequences and ChemBERTa for SMILES.\n",
    "\n",
    "Next, we will generate embeddings for the antibody heavy chain, light chain, and antigen protein sequences in our dataset using ESM-2. These embeddings capture the structural and functional properties of each protein sequence, enabling the model to learn meaningful biological representations for downstream prediction tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd177348",
   "metadata": {},
   "source": [
    "Let's first generate embeddings from protein sequences using ESM-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "67dd7254",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_SEQ_LENGTH = 1000\n",
    "\n",
    "# Function to get embeddings from protein sequences\n",
    "def get_embeddings(sequences):\n",
    "    embeddings = []\n",
    "\n",
    "    for seq in tqdm(sequences, desc=\"Generating embeddings\"):\n",
    "        inputs = tokenizer_esm(\n",
    "            seq,\n",
    "            return_tensors='pt',\n",
    "            truncation=True,\n",
    "            padding='max_length',\n",
    "            max_length=MAX_SEQ_LENGTH,\n",
    "            is_split_into_words=False\n",
    "        )\n",
    "        inputs = {k: v.to(device) for k, v in inputs.items()}\n",
    "\n",
    "        with torch.no_grad():\n",
    "            outputs = esm_model(**inputs)\n",
    "        \n",
    "        # Extract CLS token\n",
    "        cls_emb = outputs.last_hidden_state[:, 0, :].squeeze().cpu()\n",
    "        embeddings.append(cls_emb)\n",
    "\n",
    "    return embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3750d918",
   "metadata": {},
   "source": [
    "Now we can use the above function to generate embeddings for our protein sequences (heavy chains, light chains, and antigens) by passing the corresponding sequence lists as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "69770e72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating embeddings for heavy chains, light chains, and antigens: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating embeddings: 100%|██████████| 435/435 [01:25<00:00,  5.11it/s]\n",
      "Generating embeddings: 100%|██████████| 435/435 [01:25<00:00,  5.08it/s]\n",
      "Generating embeddings: 100%|██████████| 435/435 [01:30<00:00,  4.82it/s]\n"
     ]
    }
   ],
   "source": [
    "# Generate embeddings from protein sequences\n",
    "print(\"Generating embeddings for heavy chains, light chains, and antigens: \")\n",
    "heavy_embeddings = get_embeddings(heavy_chains)\n",
    "light_embeddings = get_embeddings(light_chains)\n",
    "antigen_embeddings = get_embeddings(antigens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "617720df",
   "metadata": {},
   "source": [
    "Users can uncomment the code below to save and load embeddings to save time on restart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "63a85b01",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save and load tensors separately\n",
    "# torch.save(heavy_embeddings, 'heavy_embeddings.pt')\n",
    "# torch.save(light_embeddings, 'light_embeddings.pt')\n",
    "# torch.save(antigen_embeddings, 'antigen_embeddings.pt')\n",
    "# heavy_embeddings = torch.load('heavy_embeddings.pt')\n",
    "# light_embeddings = torch.load('light_embeddings.pt')\n",
    "# antigen_embeddings = torch.load('antigen_embeddings.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8b3cb483",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_molecular_embeddings(smiles_list, max_length=512):\n",
    "    # Tokenize\n",
    "    inputs = featurizer(\n",
    "        smiles_list,\n",
    "        add_special_tokens=True,\n",
    "        truncation=True,\n",
    "        max_length=max_length,\n",
    "        padding=True,\n",
    "        return_tensors=\"pt\"\n",
    "    )\n",
    "    input_ids = inputs[\"input_ids\"].to(device)\n",
    "    attention_mask = inputs[\"attention_mask\"].to(device)\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        outputs = model(input_ids=input_ids, attention_mask=attention_mask)\n",
    "        # Mean pooling\n",
    "        last_hidden_state = outputs.last_hidden_state\n",
    "        masked_embeddings = last_hidden_state * attention_mask.unsqueeze(-1)\n",
    "        embeddings = masked_embeddings.sum(dim=1) / attention_mask.sum(dim=1, keepdim=True)\n",
    "    return embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "101a49ec",
   "metadata": {},
   "source": [
    "Now that we understand how to generate embeddings from SMILES strings, let's create embeddings for all linker and payload molecules in the dataset.\n",
    "As we have defined the function above, we can now pass our payload SMILES string to generate embeddings. This process transforms each payload molecule into a numerical vector representation that captures its chemical properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e4b3b3c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([435, 384])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "payload_embeddings = get_molecular_embeddings(payloads)\n",
    "payload_embeddings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecea3716",
   "metadata": {},
   "source": [
    "Similarly, we can get embeddings from linker smiles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1da1d8a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([435, 384])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linker_embeddings = get_molecular_embeddings(linkers)\n",
    "linker_embeddings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e8fa97",
   "metadata": {},
   "source": [
    "Let's inspect the shape of each embedding individually to better understand the tensor dimensions for every input component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "12ee5cff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Individual tensor shapes:\n",
      "Heavy: torch.Size([320])\n",
      "Light: torch.Size([320])\n",
      "Antigen: torch.Size([320])\n",
      "Payload: torch.Size([384])\n",
      "Linker: torch.Size([384])\n"
     ]
    }
   ],
   "source": [
    "print(\"Individual tensor shapes:\")\n",
    "print(f\"Heavy: {heavy_embeddings[0].shape}\")\n",
    "print(f\"Light: {light_embeddings[0].shape}\")\n",
    "print(f\"Antigen: {antigen_embeddings[0].shape}\")\n",
    "print(f\"Payload: {payload_embeddings[0].shape}\")\n",
    "print(f\"Linker: {linker_embeddings[0].shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "568a0941",
   "metadata": {},
   "source": [
    "Now that we’ve generated embeddings from all protein sequences, as well as the payload and linker SMILES, we can concatenate them, along with the standardized DAR value to form a complete feature vector for each ADC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a9665ef6",
   "metadata": {},
   "outputs": [],
   "source": [
    "adc_embeddings = []\n",
    "\n",
    "for i in range(len(heavy_embeddings)):\n",
    "    heavy = heavy_embeddings[i]\n",
    "    light = light_embeddings[i]\n",
    "    antigen = antigen_embeddings[i]\n",
    "    payload = payload_embeddings[i]\n",
    "    linker = linker_embeddings[i]\n",
    "    dar = dar_scaled[i]\n",
    "\n",
    "    # Convert to tensors and ensure 1D\n",
    "    def to_tensor_1d(emb):\n",
    "        if not isinstance(emb, torch.Tensor):\n",
    "            emb = torch.tensor(emb, dtype=torch.float32)\n",
    "        return emb.flatten()\n",
    "\n",
    "    heavy = to_tensor_1d(heavy)\n",
    "    light = to_tensor_1d(light)\n",
    "    antigen = to_tensor_1d(antigen)\n",
    "    payload = to_tensor_1d(payload)\n",
    "    linker = to_tensor_1d(linker)\n",
    "    dar = to_tensor_1d(dar)\n",
    "\n",
    "    # Concatenate all embeddings\n",
    "    full_emb = torch.cat([heavy, light, antigen, payload, linker, dar])\n",
    "    adc_embeddings.append(full_emb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fde5900",
   "metadata": {},
   "source": [
    "Lets see the shape of a single concatenated ADC embedding i.e., feature vector for one ADC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4d022520",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated 435 ADC embeddings, each of shape torch.Size([1729]). Final concatenated tensor shape: torch.Size([1729])\n"
     ]
    }
   ],
   "source": [
    "print(f\"Generated {len(adc_embeddings)} ADC embeddings, each of shape {adc_embeddings[0].shape}. Final concatenated tensor shape: {full_emb.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab78407",
   "metadata": {},
   "source": [
    "Now, we can check the shape of the full batch tensor (i.e., all ADCs stacked), where the first dimension is the number of ADCs and the second is the embedding size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "10e48436",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([435, 1729])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adc_batch_tensor = torch.stack(adc_embeddings)\n",
    "adc_batch_tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a51ec8df",
   "metadata": {},
   "source": [
    "We will be using the 100 nm label from the dataset as our target variable. Although the dataset also includes labels at 10 nm and 1000 nm, we choose the 100 nm label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "333738c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_col = \"label（100nm）\"\n",
    "labels = df[label_col].values  # 0 or 1\n",
    "\n",
    "# Convert to PyTorch tensor\n",
    "y = torch.tensor(labels, dtype=torch.float32).unsqueeze(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97f911bd",
   "metadata": {},
   "source": [
    "### Constructing DeepChem Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9b496cd",
   "metadata": {},
   "source": [
    "Now that we have our input features and labels prepared, let's convert them into DeepChem’s NumpyDataset format, which allows us to utilize DeepChem’s modeling and evaluation tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1a0d5a8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset created with 435 samples.\n",
      "Features shape: (435, 1729)\n"
     ]
    }
   ],
   "source": [
    "from deepchem.data.datasets import NumpyDataset\n",
    "import numpy as np\n",
    "\n",
    "X = adc_batch_tensor.numpy()  # Convert to numpy array\n",
    "y = labels\n",
    "y = y.astype(np.float32)  # Ensure labels are float32\n",
    "\n",
    "dataset = NumpyDataset(X= X, y= y, ids= np.arange(len(y)))\n",
    "\n",
    "print(f\"Dataset created with {len(dataset)} samples.\")\n",
    "print(f\"Features shape: {dataset.X.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cb4c924",
   "metadata": {},
   "source": [
    "Let’s print the dataset type to confirm it’s been correctly wrapped as a DeepChem NumpyDataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "064dda6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'deepchem.data.datasets.NumpyDataset'>\n",
      "<NumpyDataset X.shape: (435, 1729), y.shape: (435,), w.shape: (435,), ids: [0 1 2 ... 432 433 434], task_names: [0]>\n"
     ]
    }
   ],
   "source": [
    "print(type(dataset))\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcfcfbe2",
   "metadata": {},
   "source": [
    "Lets split our dataset into train test split using deepchem's splitters. Here we are using random splitters. Users are encouraged to explore DeepChem's other splitters like ScaffoldSplitter, IndexSplitter, and more, based on their requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "dfcbbc10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 348, Valid: 43, Test: 44\n"
     ]
    }
   ],
   "source": [
    "import deepchem as dc\n",
    "\n",
    "# Creating a RandomSplitter object\n",
    "splitter = dc.splits.RandomSplitter()\n",
    "\n",
    "# Splitting dataset into train, validation, and test datasets\n",
    "train_dataset, valid_dataset, test_dataset = splitter.train_valid_test_split(\n",
    "    dataset, \n",
    "    frac_train=0.8,\n",
    "    frac_valid=0.1,\n",
    "    frac_test=0.1 \n",
    ")\n",
    "\n",
    "print(f\"Train: {len(train_dataset)}, Valid: {len(valid_dataset)}, Test: {len(test_dataset)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "528d2b4b",
   "metadata": {},
   "source": [
    "#### Defining MLP (Multi-Layer Perceptron)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d010f58c",
   "metadata": {},
   "source": [
    "Before we move into model training, it’s important to understand the architecture of the MLP (Multi-Layer Perceptron). MLP is a type of feedforward neural network consisting of an input layer, one or more hidden layers, and an output layer. Each neuron in a layer is fully connected to every neuron in the next, allowing the model to learn complex, non-linear patterns from the data.  Other inputs include Dropout and Activation function. Dropout randomly disables a fraction of neurons during each training iteration, which forces the network to not rely too heavily on any one neuron and helps in learning more robust features, and activation function is used to specify the activation function used in the hidden layers of the model. <br>\n",
    "\n",
    "In our setup, the input layer has a dimension, which corresponds to the size of the combined embeddings. This is followed by 2 hidden layers that help the model extract deeper hierarchical features, and finally, an output layer that produces predictions. Other inputs includes a dropout rate of 0.2 and Relu activation function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86dfc5f0",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4afecd59",
   "metadata": {},
   "source": [
    "Let’s define a custom Torch model using DeepChem’s TorchModel wrapper. We'll use a simple feedforward network with two hidden layers for regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f6487092",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input dimension: 1729\n"
     ]
    }
   ],
   "source": [
    "ip_dim = adc_batch_tensor.shape[1]\n",
    "print(f\"Input dimension: {ip_dim}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0f15ecc3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.26054489135742187\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from deepchem.models.torch_models.torch_model import TorchModel\n",
    "from deepchem.models.torch_models.layers import MultilayerPerceptron\n",
    "from deepchem.models.losses import BinaryCrossEntropy\n",
    "\n",
    "class SigmoidMLP(nn.Module):\n",
    "    def __init__(self, d_input, d_output, d_hidden, dropout, activation_fn):\n",
    "        super().__init__()\n",
    "        self.mlp = MultilayerPerceptron(\n",
    "            d_input=d_input,\n",
    "            d_output=d_output,\n",
    "            d_hidden=d_hidden,\n",
    "            dropout=dropout,\n",
    "            activation_fn=activation_fn\n",
    "        )\n",
    "        \n",
    "    def forward(self, x):\n",
    "        logits = self.mlp(x)\n",
    "        return torch.sigmoid(logits)\n",
    "\n",
    "class ADCNetModel(TorchModel):\n",
    "    def __init__(self, input_dim= ip_dim, output_dim=1, hidden_dims=(1024, 256), dropout=0.2, activation_fn='relu', **kwargs):\n",
    "\n",
    "        model = SigmoidMLP(\n",
    "            d_input=input_dim,\n",
    "            d_output=output_dim,\n",
    "            d_hidden=hidden_dims,\n",
    "            dropout=dropout,\n",
    "            activation_fn=activation_fn\n",
    "        )\n",
    "        \n",
    "        super(ADCNetModel, self).__init__(model=model, loss=BinaryCrossEntropy(), **kwargs)\n",
    "\n",
    "model = ADCNetModel()\n",
    "loss = model.fit(train_dataset, nb_epoch=1000)\n",
    "print(f\"Training loss: {loss}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59289fdc",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df71abc4",
   "metadata": {},
   "source": [
    "To assess the model's performance, we use DeepChem's built-in evaluation metrics. We'll calculate ROC AUC and Accuracy, and you can explore other metrics in the deepchem.metrics module as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4b5deb9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean-roc_auc_score: 0.7603485838779956\n",
      "mean-accuracy_score: 0.7045454545454546\n"
     ]
    }
   ],
   "source": [
    "classification_metrics = [\n",
    "    dc.metrics.Metric(dc.metrics.roc_auc_score, np.mean),  \n",
    "    dc.metrics.Metric(dc.metrics.accuracy_score, np.mean),  \n",
    "]\n",
    "\n",
    "scores = model.evaluate(test_dataset, classification_metrics)\n",
    "for metric_name, score in scores.items():\n",
    "    print(f\"{metric_name}: {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c9a3f99",
   "metadata": {},
   "source": [
    "Now, let's compare our predicted values with the actual labels to evaluate how well the model is performing and gain insights into the quality of its predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6db29b92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions: [1.0, 0.5, 0.9999895095825195, 0.9968763589859009, 0.6089296340942383, 1.0, 0.9995952248573303, 1.0, 0.5, 0.5, 0.5, 0.9470304250717163, 0.5, 0.9944315552711487, 0.5, 0.5, 0.9991081357002258, 0.5, 0.5, 0.5, 1.0, 1.0, 0.5, 0.5, 0.5, 0.9999953508377075, 0.5, 1.0, 0.9999958276748657, 0.9999346733093262, 0.5335736274719238, 0.987384557723999, 1.0, 0.5, 1.0, 0.5, 0.9999969005584717, 0.99298495054245, 0.5, 0.5, 0.9999997615814209, 0.5, 0.5, 1.0]\n",
      "Ground Truth: [1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "predictions = model.predict(test_dataset)\n",
    "val_predictions = model.predict(valid_dataset)\n",
    "ground_truth = test_dataset.y\n",
    "val_ground_truth = valid_dataset.y\n",
    "\n",
    "print(\"Predictions:\", predictions.squeeze().tolist())\n",
    "print(\"Ground Truth:\", ground_truth.squeeze().tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b7ab6701",
   "metadata": {
    "vscode": {
     "languageId": "ruby"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 86.05%\n"
     ]
    }
   ],
   "source": [
    "val_outputs = model.predict(valid_dataset)\n",
    "y_val = valid_dataset.y  # Ground truth\n",
    "\n",
    "# Convert to binary predictions (equivalent to torch.round)\n",
    "predicted_labels = np.round(val_outputs.squeeze()).astype(int)  # Round to 0 or 1\n",
    "true_labels = y_val.squeeze().astype(int)\n",
    "\n",
    "# Calculate accuracy\n",
    "correct = (predicted_labels == true_labels).sum()\n",
    "total = len(true_labels)\n",
    "\n",
    "accuracy = correct / total\n",
    "print(f\"Accuracy: {accuracy * 100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f56dd95",
   "metadata": {},
   "source": [
    "Here, we have successfully built a complete pipeline using DeepChem in this notebook. This process involved preparing ADC sequence data, generating embeddings, training a custom Torch model, and evaluating its performance.\n",
    "\n",
    "Congratulations, here we conclude our first step toward applying deep learning for ADC property prediction!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a0937e3",
   "metadata": {},
   "source": [
    "## References <a name=\"references\"></a>\n",
    "\n",
    "<a name=\"1\"></a> [1] Chen, L., Li, B., Chen, Y., Lin, M., Zhang, S., Li, C., Pang, Y., & Wang, L. (2024). ADCNet: A unified framework for predicting the activity of antibody‑drug conjugates.\n",
    "\n",
    "<a name=\"2\"></a> [2] DeepChem (n.d.). Introduction to Antibody-Drug Conjugates.\n",
    "\n",
    "<a name=\"3\"></a> [3] Biopharma PEG. “The History Of ADC Drugs Development.” BiochemPEG.\n",
    "\n",
    "<a name=\"4\"></a> [4] Beck, A., Goetsch, L., Dumontet, C. et al. Strategies and challenges for the next generation of antibody–drug conjugates. Nat Rev Drug Discov 16, 315–337 (2017). https://doi.org/10.1038/nrd.2016.268\n",
    "\n",
    "<a name=\"5\"></a> [5] Shen, L. T., Sun, X. N., Chen, Z., Guo, Y., Shen, Z. Y., Song, Y., Xin, W. X., Ding, H. Y., Ma, X. Y., Xu, W. B., Zhou, W. Y., Che, J. X., Tan, L. L., Chen, L. S., Chen, S. Q., Dong, X. W., Fang, L., & Zhu, F. (2024).\n",
    "ADCdb: the database of antibody‑drug conjugates. Nucleic Acids Research, 52(D1), D1097–D1109. PMID 37831118.\n",
    "\n",
    "<a name=\"6\"></a> [6] ADCNet githubidrugLab.(2024). ADCNet: a unified framework for predicting the activity of antibody‑drug conjugates."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d4daf9d",
   "metadata": {},
   "source": [
    "# Congratulations! Time to join the Community!\n",
    "\n",
    "Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:\n",
    "\n",
    "## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)\n",
    "This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.\n",
    "\n",
    "## Join the DeepChem Discord\n",
    "The DeepChem [Discord](https://discord.gg/5d5bEVSt) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfd9308f",
   "metadata": {},
   "source": [
    "# Citing this tutorial\n",
    "If you found this tutorial useful please consider citing it using the provided BibTeX."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "431ceea7",
   "metadata": {},
   "source": [
    "```\n",
    "@manual{Molecular Machine Learning,\n",
    " title={Introduction to ADCNet: Predicting ADC Activity with DeepChem},\n",
    " organization={DeepChem},\n",
    " author={Patra, Sonali Lipsa, and Singh, Rakshit Kr. and Bisoi, Ankita and Ramsundar, Bharath}\n",
    " howpublished = {\\url{https://github.com/deepchem/deepchem/blob/master/examples/tutorials/ADCNet.ipynb}},\n",
    " year={2025},\n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adcnet",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
